REMARKS 

Claims 1-30 are pending in the application. Claims 1-30 stand rejected. 

Applicant respectfully requests reconsideration in view of the foregoing 
amendments and the remarks hereinbelow. 
Rejection of Claims under 35 U.S.C. 103: 

Claims 1-30 stand rejected under 35 U.S.C. 103(a) as being unpatentable 
over Borovoy et al. (U.S. 5,873,107). 

I. Claims 1-12: 

Borovoy et al. generally describes an information retrieval system that is 
adapted extract keywords from text entered by the author and to retrieve reference 
materials from a database having similar keywords associated therewith The 
purpose of Borovoy et al. is to provide relevant research information to an author 
during generation of a main document or during the pendency of a text authoring 
session, e.g. an on-line chat session. The user will have the ability to observe 
obtained research information in a convenient manner while the author is 
preparing text that refers to such information. This saves the author time and 
prevents unnecessary interruptions in the authoring process that are required to 
access the research information during the authoring process. 

The present invention provides a different function as is described in 

greater detail in claim 1 : 

1. A method for integrated retrieval and annotation of stored 
images involving a computer application in which text is input by a 
user, said method comprising the steps of: 

a) providing metadata associated with the stored images; 

b) running a user application in which text is entered by a user; 

c) continuously monitoring text typed by the user into the 
application to isolate the context expressed by the text; 

d) matching the context with the metadata associated with the 
stored images, thereby providing one or more matched images; 

e) retrieving and displaying the matched images; and 

f) utilizing the context to provide suggested annotations to the 
user for the matched images, together with the capability of selecting 
certain of the suggested annotations for subsequent association with the 
matched images. 

Using such a system, a user of a collection of stored images can type in 
text, and the context of the words being entered by the user can be identified. The 
context is used for two purposes, to retrieve images based upon the context and to 
provide suggested annotations for the retrieved images. The user can rapidly 
associate relevant annotations to the retrieved images. Such a result is not 
described or suggested by Borovoy et al. 

-8- 



1. Borovoy et al. does not teach or suggest editing or otherwise 
annotating the reference documents. 

Borovoy et al. makes no suggestion that there is a need or a desirability to 
stop editing the main document from which the keywords were extracted and, 
instead, edit the references themselves. The Office Action of January 27, 2005 
admits this stating: 

the reference does not explicitly disclose utilizing the context to provide 
suggested annotations to the user for the matched images, together with 
the capability of selecting certain of the suggested annotations for 
subsequent association with the matched images. 

Therefore, these features must be obvious to one of ordinary skill in view 
of Borovoy et al. alone. The Office Action then posits the following two-part 
logical construct in support of such a finding of obviousness: 

A. however, the reference provides the variant form of the 
invention utilizing the retrieval of images based upon the author 's 
text The author which input text, which then causes the image 
retrieval application to begin, followed by the user having the 
ability to manipulate said image The user would have the decision 
of what to do with that image data, but the invention provides the 
suggestion of manipulating data retrieved by the application. 

The Borovoy et al. reference states only the following with respect to 

obtaining images; 

"Still further, note that the present invention is equally 
applicable to searching retrieving documents or files which are not 
strictly text based. Because audio and image based files typically 
have text based file names or other descriptive information 
associated with them, they can likewise be searched for and 
retrieve. One can imagine numerous possibilities where, for 
example, an author is writing about a particular subject and the 
retrieved material includes relevant still images, graphics, or video 
clips or even a file containing a voice recording which has been 
"read" by a speech recognition dictation engine and can thus be 
determined to be relevant." Column 6, Lines 33-44 

Accordingly, Borovoy et al. simply does not teach or suggest editing of 
obtained images. Instead Borovoy et al. suggests only that such references are 
displayed for use in editing the main document. It can be argued that it is obvious 
from this that data from such reference documents can be used in the main 
document and even edited as a part of the main document as is suggested in the 
Office Action. However, it is not obvious from Borovoy et al. to do anything 
more than this as Borovoy et al. provides no motivation to do anything other than 
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edit a main document using information from the references. Only the 

impermissible use of hindsight would lead one to conclude that it is obvious to 

use Borovoy et al. for some other purpose and in particular to a purpose that is 

contrary to the stated purpose of Borovoy et al. which is to allow editing of a 

main document without interruption. 

B. Therefore it would have been obvious to one of ordinary 
skill in the art at time the invention was made to have modified the 
image retrieval methods taught by Borovoy and included feature of 
editing the retrieved image and improving an author's formal 
document research based on past works. 

This is not what is claimed. What is claimed is the steps of e) retrieving 
and displaying the matched images; and f) utilizing the context to provide 
suggested annotations to the user for the matched images, together with the 
capability of selecting certain of the suggested annotations for subsequent 
association with the matched images. 
Step F therefore includes two parts: 

1) utilizing the context to provide suggested annotations to the user 
for the matched images, together with 

2) selecting certain of the suggested annotations for subsequent 
association with the matched images. 

The combination cited in the office action fails to provide either of the 

parts of step f) for each of the following three reasons: 

2. Borovoy et al. Depends Upon Keyword Searching and Does Not 
Search for Reference Documents Using Context. 

Borovoy et al. does not determine use context of the extracted keywords 

for any purpose. Instead, the only suggested use of the word "context" in 
Borovoy et al. is the use in FIG. 1 of the word context in reference to a display 
309 in which reference documents are presented in visual context with the 
portions of the main document that contain the text from which the keywords 
were extracted, (see Borovoy et al., FIG. 1, column 5, lines 1-10.) Borovoy et al. 
teaches only that the keywords obtained from the main document are to be used 
for use in formulating search queries. While Borovoy et al. suggests that a range 
of different search tools can be applied to the keywords, it is the keywords, that 
form the core of the search. Borovoy et al. fails to teach or suggest determining 
context based upon the keywords, conducting searches based upon the context. 

The use of context for searching provides a significant advantage over the 
use of keywords as can be determined by how a keyword search will search when 
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the word "yesterday" is searched. A keyword search using the word yesterday 
will search for the word "yesterday", and, possibly words similar to the word 
"yesterday 5 '. A context search will search for images that were created or edited 
on the day previous to the day that the word "yesterday" is entered. This yields 
substantially different search results. 

3. Borovoy et aL Fails to Teach the Suggesting of Annotations for 
Annotating Reference Documents Based Upon the Context 

Nothing in Borovoy et al. suggests that any annotations are made based on 
the context. For example, to the extent that the Office Action contends that 
Borovoy et al. uses the keywords to obtain reference documents, and to the extent 
that the Office Action is correct in the assertion that it follows from this that such 
reference documents that can be edited, the Office Action fails to show how doing 
so would, in addition, utilize the context to provide suggested annotations to the 
user for the matched images. First, Borovoy et al. cannot do so as Borovoy et al. 
fails to provide for the use of context and further it cannot do so because Borovoy 
et al. fails to provide any description of any use of keywords other than obtaining 
reference documents. 

Further, if the reference documents themselves are opened as a main 
document in Borovoy et al. as suggested in the Office Action, then a new set of 
references will be searched for and appear based upon new keywords in the newly 
opened main document. Even if the new references are could be considered as 
potential annotations to the opened reference document, it cannot be said that 
context from the original text is used for determining suggested annotations. 

Instead, in the present invention, context determined from the words 
entered by the user is used both for obtaining reference documents and for 
determining suggested annotations. 

4. Borovoy et aL Fails To Suggest the Capability of Selecting 
Annotations for Subsequent Association with Matched Images. 

Assuming only for the purpose of argument that Borovoy et al. described a 
system that provided such proposed annotations, Borovoy et al. would still fail to 
provide any teaching of the capability of selecting certain of the suggested 
annotations for subsequent association with the matched images. Specifically, 
Borovoy et al. fails to provide any structure for selecting annotations for 
association with images that match the context. 

Further, Borovoy et al. also fails to describe a system that provides the 
ability to associate any data with "images" instead, assuming only for the purpose 
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of argument that Borovoy et al. describes a system that can allow for the editing 
of a reference document, such editing could only be performed by opening each 
reference document individually and associating data therewith. However, as 
claimed, the present invention is capable of associating a selected annotation with 
the "matched images". 

For these reasons the applicants respectfully request reconsideration claim 1 
and prompt allowance of claim 1 and all claims that depend therefrom. 
II. Claims 13-21: 

Independent claim 13 stands rejected on grounds that 'the limitations 
reflect the instructions using performing similar methods as those claim in 
independent claim 1, and further view of the following, is rejected along the same 
rationale. Furthermore, the reference teaches that that documents being author 
may be as extensive and formal as a daily journal entry for e-mail message 
(compared to "running an e-mail application in which text is entered by a user 
into a message window"). See column 6, lines 25-28." 

Claim 13 has been cancelled and claim 14 now stands in independent 

form. As amended, claim 14 claims: 

14. Tho method as claimed in claim 1 3 further 
comprising tho stop of A method for utilizing images that are 
stored in a data base with an e-mail application in which text is 
input by a user into a message, said method comprising the steps 
of: 

a") providing metadata associated with the stored 

images: 

b) running an e-mail application in which text is 
entered by a user into a message window; 

c) continuously monitoring text typed by the user 
into the message window to isolate keywords in the text; 

d) matching and ranking the keywords with the 
metadata associated with the stored images, thereby providing one 
or more matched images that are ranked according to the 
keywords: 

o) retrieving and displaying the matched images 
alongside the message window in order of their rank: and 

f) providing the user with the capability of moving 
selected ones of the matched images into the message window and 
utilizing the context to automatically provide new keyword 
annotations for the matched images. 
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Claim 15, has also been amended and stands in independent form. Claim 
15 as amended, is not taught or suggested by Borovoy et al. generally for the 
reasons stated above with respect to claim 1. 

Claims 16, and 18-21 have also been amended and now depend from 
claim 14, accordingly the limitations of these claims are not believed to be taught 
or suggested by Borovoy et al. generally for the reasons stated above with respect 
to claim 1 . Further, Borovoy et al. does not teach or suggest the limitations of 
claim 16 in that nothing in Borovoy et al. describes an automatic method for 
updating metadata associated with a reference document. Instead, all editing or 
annotation in Borovoy et al. is performed manually. 

Further, Borovoy et al. does not teach or suggest the limitations of claim 
17 in that nothing in Borovoy et al. describes an automatic method for updating 
metadata associated with a reference document. Instead, all editing or annotation 
in Borovoy et al. is performed manually. 

III. Claims 22-30: 

Claims 22 - 30 have been rejected on grounds that "the limitations reflect 
the system comprising instructions used for performing the methods as claimed in 
1-13, and in further view of the following, are rejected along the same rationale." 

Claim 22 stands in independent form and claims the following: 

22. An autonomous agent for use in automatically 
interfacing a text-based application with a picture archive 
containing representations of images and metadata associated with 
the images, said agent comprising: 

a language analyzer for continuously monitoring text 
entered by the user into the application and extracting keywords 
appropriate to the context surrounding the entered text; 

a retrieval agent for retrieving images from the archive on 
the basis of one or more matches between the extracted keywords 
and the metadata; 

an annotation agent for providing suggested annotations to 
the user for the matched images based on the extracted keywords, 
together with the capability of selecting certain of the suggested 
annotations for subsequent association with the matched images; 
and 

a picture database viewer for displaying the retrieved 
images to the user and for enabling the images to be incorporated 
into the application, as desired by the user. 

While Borovoy et al. discloses a system wherein keywords are detected in 
entered text, Borovoy et al. fails to disclose a language analyzer that monitors 
such text and extracts keywords that are appropriate to the context surrounding 
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the entered text. Instead, Borovoy et al. only discloses detecting keywords in the 
text and searching according to the keywords as discussed generally above. 
Accordingly, to the extent that Borovoy et al. discloses any step of obtaining 
images in response to detecting keywords, the keywords used by Borovoy et al. 
do not contemplate and are unaffected by the context of the text surrounding 
entered text. To the extent that Borovoy et al. can be said to suggest obtaining 
images, the images that are retrieved by Borovoy et al. will be different from the 
images received by the retrieval agent of claim 22. Thus, Borovoy cannot meet 
the limitations of claim 22 in that it does not provide the language analyzer as 
claimed. 

Borovoy et al. further lacks an annotation agent. As discussed in greater 
detail above, Borovoy et aL arguably discloses a system wherein keywords, 
unaffected by context, can be used for searching purposes. However, nothing in 
Borovoy et al. suggests any further use of such keywords. In particular, Borovoy 
et al. fails to show any system wherein an annotation agent provides suggested 
annotations to the user for the matched images that are obtained based upon the 
extracted keywords. The only purpose of the keywords in Borovoy et al. is to 
obtain references. Simply stated, Borovoy et al. lacks any teaching or suggestion 
of using context to select certain suggested annotations for subsequent association 
with the obtained references. 

Assuming only for the purpose of argument that one could establish that 
Borovoy et al. provides for the ability to access one of the reference documents so 
that the reference document itself could be edited, the only disclosed means for 
doing so would be to open the reference document as a main document as 
indicated in the Office Action. As noted above, as soon as this is done, the set of 
keywords in the main document upon which the reference document was 
originally obtained will be replaced by new keywords associated with the 
reference document that has been opened as a main document and a new set of 
reference documents will be obtained using the new keywords. Thus, there would 
never be a situation wherein the set of keywords extracted from the main 
document are used for both the purpose of obtaining reference documents, and for 
determining suggested annotations that can be associated with the obtained 
reference document. 

Accordingly, claim 22 and all claims that depend therefrom are believed to 
be in a condition for allowance. 
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It is respectfully submitted, therefore, that in view of the above 
amendments and remarks, that this application is now in condition for allowance, 
prompt notice of which is earnestly solicited. 



Roland R. Schindler II/jrk 
Rochester, NY 14650 
Telephone: 585-588-2736 
Facsimile: 585-477-1148 



Respectfully submitted, 





attorney for Applicant(s) 
Registration No. 40,802 
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